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ABSTRACT 

Professor Stake feels that Professor Glaser's paper 
does not give adequate attention to how descriptions and judgements 
of the instructional process can be used to facilitate educational 
decisions. He also asserts that behavioral specification, as it is 
usually defined, is often impractical and unnecessary, and he 
challenges the primacy of goal specification as a first logical step 
in instruction or evaluation. Objectives must be reported, but 
attention must be given to how they change over time and the priority 
given by different groups to different objectives. (Author) 
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COMMENTS ON PROFESSOR GLASER'S PAPER ENTITLED 
"EVALUATION OF INSTRUCTION AND CHANGING EDUCATIONAL MODELS" 

Robert Stake 

I received a draft of Bob Glaser's paper several weeks ago. 
There was postage due. Bob had missed the mark by a factor of .64. 
On payment of the additional 28 cents, I reflected on the possi- 
bility of this being an omen: Was he going to miss the mark on 

his theory of the evaluation of instruction as well? 

I think he did. 

Let me refresh your memories. He presented six components of 
a model of instruction. As I understand them, they were the speci- 
fication of objectives, the diagnosis of entry behavior, the selec- 
tion of instructional procedures, the surveillance of instruction, 
the modification of instruction, and the modification of the system. 
He spotted needs for measurement and evaluation within each of 
the six. He claimed that the needs for measurement and evaluation 
were fluid, subject today particularly to three forces: increased 

emphasis on cultivation of skill and understanding of the basic 
discipline, increased emphasis on the continuity of education 
through the life span, and increased emphasis on individual 
instruction, passing it from the ideal to reality. 

Bob presented a model of instruction and discussed some of 
the implications for evaluation. He did not develop a model or a 
theory for the evaluation of instruction. Earlier Ben Bloom 
talked about evaluation within a theory of testing; Bob Glaser 
talked about evaluation within a theory of instruction, not a theory 
of evaluation. Perhaps if Merl Wittrock had given them behavioral 
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specifications, we would have covered by now a theory of evalua- 
tion. I am not claiming that he should have. I am just saying 
that in spite of the original titles of the morning's paper, we 
have not yet considered a model or theory of evaluation in the 
sense that either Ben Bloom or Gene Glass define it. 

Bob Glaser indicated six functions that require extensive 
measurement. There are many responsibilities for student testing, 
there are observations to be made, descriptions to be recorded, 
judgments to be set, decisions to be made. Instruction requires 
all of these. Evaluation of instruction, as separate from evalua- 
tion within instruction, requires them, also. Bob Glaser has not 
told us how the descriptions and judgments of the instructional 
process might be fitted together to provide us with the infor- 
mation we need to make educational decisions. 

Repeatedly Bob referred to his system, or the phenomenon or 
the task, as one of individual student learning. This is an 
important focus. With Maurie Lindvall and other colleagues at 
the University of Pittsburgh Oakleaf School, Bob has helped us to 
realize how, in this day of large student bodies, we can individ- 
ualize instruction. But our concern is the evaluation of instruc- 
tional systems. 

This evaluation is considerably more than the aggregate of 
assessment of progress for all children at all levels. Here the 
unit of analysis is not the child but the classroom, the school 
system, or the nation. The satisfaction we get from its operation 
is the focus. The changes we can make to increase our satisfac- 
tion with the system are our focus. It is a different focus. 
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Where we might have expected him to discuss evaluation of 
the system (in his sixth section) , we found only a mention of the 
need for evaluation, plus an inspirational quote from Lee Cronbach. 
Bob did not consider any of the more recent literature on evalua- 
tion. What of Mike Scriven? What of Michaels and Metfessel? What 
of the writings which have attempted to apply operations analysis, 

cost-benefit analysis, and decision theory to the instructional 
system? 

Perhaps I am wrong in expecting Bob Glaser (and many of my 
colleagues specializing, in measurement) to expand the concept of 
evaluation beyond ascertainment of how well objectives have been 
accomplished. Perhaps it would be more useful to postpone the 
call for a broader definition until we are doing a satisfactory 
job of a more limited kind. But I think not. It is important to 
help today's educator discover the worth of instruction. Now 

is the time for the technologist to come to the aid of his col- 
leagues . 

There are many points in the paper with which I enthusiastically 
agree. I warmed with satisfaction on hearing Bob’s declaration 
that it is worthwhile trying to learn the nature and judge the 
quality of the teaching, whether or not we can measure the extent 
and judge the quality of the resulting student content. I was 
pleased to hear his emphasis on the need for continuous, automatic 
correcting and system-wide correcting techniques for evaluation. 

Bob's concern for interaction is well placed. Although I am 
not encouraged by the research results from Larry Stolurow, Jacob 
Beard, and others, I believe that we should continue to anticipate 
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interaction between instructional methods and student characteris- 
tics. If there are important accommodations of instruction to 
children, in addition to those in the area of pacing and step size 

we should learn the additional real and indirect cost of individ- 
ualization. 

There are many important bits and pieces of the paper that I 
would like to discuss at length. But I feel I should use the 
remaining time on just one. 

Bob Glaser spoke of the "now platitudinous assertion" of the 
necessity for behavioral objectives. "Platitudinous" means "dull" 
and insipid," and I wouldn’t have been so uncharitable as to call 
the assertion that. It also means "commonplace." That it is. It 
does not mean "no longer needing validation." I challenge the 
validity of the role that he gives to behavioral objectives. 

I object to his statement that the first responsibility in 
evaluation is the specification of objectives. Much of what he 
said in the paper conveys the conviction that student-outcome 
behavior should be specif ied- - that is, committed early to formal 
language, English language. He supports those who say that all 
human accomplishment can be objectively, operationally described. 

So do I. He also says, as a regular initial step in instructional 
evaluation, objectives should be formally stated. For some large- 
scale instructional projects, I agree. For classroom evaluation, 

I do not. I claim that it is impractical, unnecessary, and dis- 
tractive to develop and evaluate curricula with high reliance on 
behavioral objectives. Analysis does not necessarily aid perform- 
ance. A designer of "paint by numbers" kits might point out, as 
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I believe information theorist Claude Shannon has, that any paint- 
ing is essentially a collection of areas of discernably different 
solid colors. It does not follow that painting by numbers is the 
best way to paint a Mona Lisa. 

Few of you in the audience, in your creative, scholarly work, 
operate from a table of specified objectives. The quality of paint 
ing or researching or instruction we are getting from today's best 
specifications is greatly inferior, as I see it, to what we can get 
from the best craftsmen. Professional education does not have the 
talent for specifying objectives. It does not make sense to me 
to base evaluation plans on a talent we do not have, nor can afford 
The price of gaining and using thi 5 talent, I claim, is one hundred 
times the price of the instruction itself. Impractical, I say. 

I am not saying behavioral specification will not work be- 
cause teaching is an art rather than a science. I am saying behav- 
ioral specification will not work because teachers are artists 
and artisans, not linguists and philosophers; and it costs too 

much for us technologists to talk to them in a language they do not 
understand. 

It is right for us to continue to develop a technology of 
education. Our plans for a technology should be practical. The 
technology we build should not require a revolution but should be 
realistic, designed with an eye to the cost of institutionaliza- 
tion. As much as possible, we should capitalize on present teach- 
ing skills and commitments. To insist on behavioral specifications 
is to ask for new evaluation skills and commitments. I believe 
that a high quality of instruction and a high quality of evaluation 
can be developed without this specification. 



There are other ways of representing objectives. Obviously, 
we can use examples. An educator, by a nod of ths head, can indi- 
cate which representation of behaviors is close to his ideals. 

We can film behaviors. We can describe them in anecdotal form. 

We can use test items to represent goals . There are alternate ways 
of specifying objectives, further, perhaps, from some true intent 
that lies within the mind or heart of an educator; but these are 
specifications that are within our budget of talent and patience. 
With them we can get along without formal goal statements. 

Some of these are not popular notions with some of you, I know. 
It may be only your politeness and pity that quiets your irrita- 
tion. Obviously I am not criticizing Bob Glaser, but the Learning 
Research Center, the Center for the Study of Evaluation of Instruc- 
tional Programs, the Southwest Regional Laboratory, the National 
Society for Programmed Instruction, and all those Camelots of behav- 
ioral linguistics. 

My first two points are that behavioral specification, as we 
usually define it, is often impractical and unnecessary. I will 
now challenge the placement of goal specification as a first 
logical step. Bob and others have agreed that you do not do all 
your goal specification first. You modify as you go along. So, 
we agree on that. The point I want to make is that there is a 
still prior logical step that Bob did not mention. Gene Glass has 
already referred to it. We need to consider the vital question of 
goal competition and goal selection. Goal selection is an integral 
part of instruction. We psychologists are not without qualification 
for dealing with preferences and priorities. Yet we have, I believe, 
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avoided incorporating goal selection into our evaluation technology 
so far. 

The other day I refreshed my impressions of the difference 
between the goal domain of the programmer and the goal domain of 
the classroom teacher. I looked over the programmed text on 
population genetics that Dick Anderson and his lab workers wrote 
for the BSCS biology project. I also visited a biology class at 
the University High School in Urbana. Worlds apart! 

Both were smooth, provocative, response -oriented operations 
but were seeking quite different responses. The program writers 
knew what specific responses they wanted. The teacher did not. 

The aim of the classroom teacher was to provide opportunity for 
reflection and reaction which she could use in an operant condi- 
tioning paradigm. She was willing to pick up, reinforce, and 
build upon many kinds of responses. More generally, may I say 
that our qualified classroom teachers seek a response with poten 
tial for development. They scan hundreds of desired responses 
to find any one of many desired responses. Sometimes the ideal 
responses can be elicited directly; many times not. Some responses 
are suspect when directly elicited. The skilled teacher has 
learned to recognize a set of desired behaviors and to reinforce 
them and to shape them when they occur. The context is deemed 
very important. What is the classroom situation/ What is the 
student’s situation? The presence of these responses in terms of 
a problem context, in an affective context, is very important. 

Bob Glaser made the point that there can be important inter- 
actions between student type and teaching method. These teachers 
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show me that there can be important interaction between student 
readiness and teaching method. We need more research on that. 

The skilled teacher has many objectives to work from. She is not 
very concerned about getting them all done. She says, "There's 
another teacher coming along next year, or next hour." Each one 
will seize an opportunity to do particular objectives well. They 
will reassign priorities to goals on the spot. This reconsidera- 
tion of priorities is the important purchase that we make when we 
choose skilled teachers rather than programmed material. 

The educational evaluator must deal with priorities. To give 
priorities to goals, he must know what those goals are. But the 
goals of educators increase and grow. The evaluator cannot avoid 
early definition of general goals, but he cannot ignore changing 
goals and changing goal priorities. As a first logical step, the 
instructional specialist may consider how to define and opera- 
tionalize objectives. As a prior logical step, the curriculum 
and evaluation specialists should consider how to define and give 
priorities to objectives. 

i 

I 

| Selection of goals and revision of priorities are important 

components of instruction. They belong in a model of instruction; 

I 

I 

they belong also in a model of evaluation. Our technology should 

j 

not ignore the fact that educators need more rational and objective 

procedures for specifying the competing priorities of different 

I 

j instructional goals . 

| 

The mark of evaluation is information for decision-making. 

It is more than a by-product of good instruction. I think Bob 
owes us another 28 cents worth. 




